feat: central graph store (graphify remote init/push/pull/delete) — graphify-out links into a store folder + repo-local sync hooks#1752
Open
muthuishere wants to merge 2 commits into
Conversation
31729da to
45f075f
Compare
45f075f to
3a7de52
Compare
bcbdc06 to
caf0e67
Compare
24f3ad1 to
8390b30
Compare
…repo-local sync hooks (graphify remote)
A committed .graphify/config.json ({"store": "~/graphify-store"}) makes every
graphify run keep ./graphify-out as a link (POSIX symlink / Windows directory
junction, no admin) into <store>/<repo>/<branch>/<module-relpath>/graphify-out.
Writes physically land outside the repo; every literal graphify-out/... path —
CLI defaults, the agent skill, plain cat — keeps working through the link, so
the skill and skillgen artifacts stay byte-identical to upstream.
One command group: graphify remote <init|push|pull|delete>
- remote init: bootstrap the committed .graphify/ (config.json + starter S3
push/pull hooks; swap for .sh/.js/.ps1/.cmd of the same name or point config
{"push": ...} anywhere; creds stay in env, never in the repo)
- remote push/pull: run the repo's hook with GRAPHIFY_STORE_DIR/PREFIX/CONFIG
in the env (resolution: config path -> .graphify/<action>.* ->
~/.config/graphify/hooks/); pull then fans a link out for every module in
the store — one pull recreates a fresh clone's whole working set
- remote delete: leave the store — every link becomes a real local folder
holding a copy of its store data (store untouched)
Adoption/exit are both safe: pre-existing local graphify-out dirs (even ones
tracked in git) migrate into the store once; a bare 'graphify-out' .gitignore
entry is maintained (trailing-slash misses symlinks; bare also stops git
descending junctions on Windows). Repo key from the origin remote basename so
all clones share it; branch switches retarget the link. No config -> behaviour
identical to upstream.
8390b30 to
c8b1268
Compare
… segments The store path is now the whole key: graphify-out links into <store>/<module>/graphify-out with no <repo>/<branch> segments. A repo points at the same store location on every branch (no retarget, no rebuild; `graphify update` refreshes), and each repo just names its own store path — `remote init` defaults it to ~/graphify-store/<folder>. Removes the origin-URL parsing and branch derivation. Hook env: GRAPHIFY_STORE_DIR is the <store> tree, GRAPHIFY_PREFIX the store basename; GRAPHIFY_REPO/GRAPHIFY_BRANCH removed. Docs + 48 tests updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Teams with large graphs (hundreds of MB per module, monorepos with many modules) need
graphify-outout of git and shared through a central place. This PR adds that without touching the skill: a committed.graphify/config.jsonselects a store folder, graphify keeps every./graphify-outas a link into it, andpush/pullhooks committed in the repo sync the store anywhere (S3, rsync, git-lfs, a network share).One command group:
Why a link (and not path rewriting)
The skill, the CLI defaults, and users' habits all use the literal
graphify-out/...path (75 references inskill.mdalone). A link keeps every one of those working unchanged while the bytes live outside the repo —graphify/skill*.md,graphify/skills/, andtools/skillgen/are byte-identical tov8in this PR (--check: 134 artifacts OK). The filesystem does the redirection, so the CLI and the skill can never disagree.Behaviour
lstatwhen already correct). No config → identical to today.<store>/<module-relpath>/graphify-out— POSIX symlink, Windows directory junction (no admin rights).<repo>/<branch>segment. The repo points at the same store location on every branch (switch branches → no retarget, no rebuild;graphify updaterefreshes). Each repo names its own store path in its config;remote initdefaults it to~/graphify-store/<folder>, editable to anything.graphify-out/— even one tracked in git — is migrated into the store once; committing then records the graph files leaving git.remote deletereplaces each link with a real folder holding a copy of its store data (the store is untouched), so a team can adopt and later go back. Both directions verified live..gitignoregets a baregraphify-outentry in the same step (bare on purpose:graphify-out/matches only directories — git sees a symlink as a file, and on Windows git would descend into an unignored junction). Even unignored, a symlink commits as a ~100-byte path, never data.graphify-outis never clobbered (warns and skips).GRAPHIFY_OUT(absolute) still overrides everything and disables linking.Sync hooks
remote push/pullrun a hook — Python, Node, shell, PowerShell,.cmd/.bat, or any executable. Resolution: config"push"/"pull"path →.graphify/<action>.*committed in the repo. Repo-local hooks mean a clone carries its own sync behaviour — zero per-machine setup; secrets stay in the environment (~/.aws, env vars), never in the repo.The hook gets
GRAPHIFY_STORE_DIR(the<store>tree),GRAPHIFY_PREFIX(the store folder basename),GRAPHIFY_CONFIG,GRAPHIFY_STORE,GRAPHIFY_REPO_ROOT, and the action in the env and mirrors that tree; graphify has no backend dependency. An executable hook runs directly (shebang wins); otherwise the interpreter is picked by extension, so it also works on Windows.remote initscaffolds working S3/MinIO starters (boto3 viauvshebang;cache/excluded as a local-only accelerator) with header comments telling teams how to swap languages.remote pullalso fans a link out for every module found in the store, so one pull on a fresh clone recreates the whole working set. If the store folder is already shared (NFS/Dropbox), push/pull are unnecessary.Files
graphify/store.py(new) — config discovery, store keying, link/junction creation, migration, materialization, gitignore upkeepgraphify/remote.py(new) — thegraphify remote <init|push|pull|delete>group: hook resolution + invocation, bootstrap, exitgraphify/remote_hook_templates.py(new) — S3 starter hooksgraphify/__main__.py/graphify/cli.py— dispatch + helpdocs/graph-store.md(follows the existingdocs/feature-doc convention),tests/test_remote_sync.py(49 offline unit tests)Testing
pytest tests/→ 3049 passed (Python 3.12),python -m tools.skillgen --check→ OK, ruff/pyright clean.~expansion, the store path as the sole key), linking (idempotency, branch-independence, dangling-link self-heal, customGRAPHIFY_OUTnames, file-collision safety), migration (including collisions — local wins), gitignore variants (bare/slash/**, missing trailing newline), hook resolution (auto-discovery without a config key, extension priority, all six interpreters, executable/shebang path), the full hook env contract,remotedispatch incl. thedeinitalias, init bootstrap/idempotency/other-language respect, materialization, and the S3 template'scache/exclusion executed offline.git statusclean; branch switch keeps the same link (branch-independent, no rebuild); publisherremote push+ fresh-clone teammateremote pullverified against a real S3 bucket (Hetzner Object Storage) with the scaffolded boto3 hooks (6 objects round-tripped under one flat<store>prefix, cache excluded). Adoption + exit scenario verified live with graphs previously committed to git (2 tracked files → 0 → 2).Closes #1751